Is this an expensive operation that exists mostly for debugging? It seem unusual what we rely on a log file. If this operation should be used sparingly, it would be good to mention thus,

This is the shortcoming to use stunnel. Although stunnel is a reliable tool to proxy TLS, it's hard to get the error like the native ssl lib. There is no programmatic API or formatted error code in stunnel. While replacing stunnel in our repo is really a big project. So it is the only way for us to get certificate checking error via stunnel log.
I'm clear about the fragility, so I create the new file stunnel_log_scanner to handle this, with real-world stunnel log in unit test.
I don't think the diagnose is a expensive operation, comparing to the network event. When reading log, the input channel uses buffered I/O - doesn't make a system call for each line. It also uses lseek to jump to a specific position (avoids re-reading from beginning).

Logfiles get rotated, renamed, and compressed. How does this interact with this module?

Don't worry about it. The log to check is not the secure.log in /var/log. It's in forkexecd data dir(stunnel process is created by forkexecd) when stunnel is running. So no rotate, rename and compress.

Thanks. I'm fine with this.

The in_channel remembers the position already. Could the ic be used here directly, instead of using a integer for the position and re-open the log file at each time of checking the log?

Good idea. Let me try.

It's worth mentioning that the input_line may get a partial line when the stunnel pauses writing the remaining part of the line; and hence the checker may miss the signature forever.

Yes. I have considered this issue. On most normal cases, input_line read untill the new line. In rare case, it reads to End_of_file, but the line is patially flush to the file. Generally the rare case should be caused by stunnel crash which can't be handled.
From the other hand, at the moment the user call diagnose, if the log file ends at "certificate veri" with no new line, the certificate error is actually ignored. But in the user scenario, the diagnose only be called after their network event fails once and then return. Even we read from the start of the log file, it doesn't help the case.
So I think we needn't consider the rare case.

For example, when max_retries is 2, it checks 4 times.

check 0
-> 0 <= 2 -> check  1
             -> 1 <= 2 -> check 2
                          -> 2 <= 2 -> check 3
                                       -> 3 <= 2 -> stop

From the matching's point of view, there would be only two results: Found (Some ...) or Not_found (None).
And for the Found, the caller can determine it's an expected good result or an error. Something like:

type scan_result = (string, string) Result.t  option

let find ~sigs ~box line =
  sigs
  |> List.find_map (
    (fun affix ->
      if Astring.String.is_infix ~affix line then
        Some (box line)
      else
        None
    )

let check_good good = find ~sigs:good ~box:Result.ok

let check_bad bad = find ~sigs:bad ~box:Result.error

let check_both ~good ~bad line = (check_good good) >>= (check_bad bad)

let check_log ~ic ~line_checker ~no_match=
  let rec loop () =
    match input_line ic with
    | line -> (
        match line_checker line with
        | None ->
            loop ()
        | Some r ->
            r
    )
    | exception End_of_file ->
        no_match
  in
  loop ()